Every system that provides services to clients needs to protect itself from a crushing load of service requests that could potentially overload the system. In general, for a Web service or remote procedure call (RPC) service, a system is considered to be in an “overloaded” state if it is not able to provide the expected quality of service for some portion of client requests it receives. Common solutions applied by overloaded systems include denying service to clients or throttling a certain number of incoming requests until the systems get out of an overloaded state.
For example, a naïve solution may throttle incoming requests if the system is overloaded and may stop throttling once the system gets out of overloaded state. This implementation, however, can lead to an oscillatory behavior in which the system is able to exit an overloaded state by throttling incoming requests, but gets right back into the overloaded state once it removes the throttle.
Many current systems avoid an overload scenario by comparing the request rate and/or the quality of service perceived by the system itself with a fixed or varying global threshold and selectively refusing service to clients once this threshold has been crossed. However this approach does not take into account differences in the expectations of different clients regarding quality of service. In addition, it is difficult, if not impossible, to define a single global threshold that is meaningful (much less that provides acceptable performance) in a system that receives different types of requests at varying, unpredictable rates.
While the technology described herein is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the drawings and detailed description thereto are not intended to limit the disclosure to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the present disclosure as defined by the appended claims.